RL Training-Inference Inconsistency, Diagnostics, and Solution
Training-Inference Inconsistency refers to a class of issues that the logprob of tokens mismatch between inference stage and training stage, causing training problems like reward dropping, etc. Results could be amplified in case of MoE training. This post briefly discusses related problems, diagnostics, and possible causes.
Table of Contents
1. Why Industrial RL Frameworks Adopt Different Engines for Inference and Training1
A preliminary knowledge, currently in all industrial frameworks adopts different engines for inference and training, for example, SGLang for inference and Megatron for training in Slime, vLLM for inference and FSDP2 for training in VeRL.
Why this design? This is because the requirements of inference and training are different. Training requires backprop, large HBM, gradient computation, and parameter synchronization (if parallelized), while inference requires high throughput, low latency, KV cache management, and dynamic batching, as in vLLM and SGLang. They have different optimal objective. Therefore, currently there’s no single framework can handle these 2 as the same time.
2. What is Training-Inference Mismatch
- Training-Inference Mismatch
- This is a class of issues caused by the numerical difference during LLM RL post-training between the inference engine (e.g., vLLM, SGLang) and the training engine (e.g., FSDP, Megatron).
2.1. Training-Inference Mismatch in MoE Models
The TIM problems get more severe when it comes to MoE models, for a couple of reasons.